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PROJECT SUMMARY 

As the responsible agency for administering the U.S. Department of 
Agriculture's Food Stamp Program, a major priority for the Food and Nutrition 
Service (FNS) is to reduce losses to the program due to fraud and error. It 
has become increasingly difficult to develop strategies that are effective in 
reducing errors. As the error rate decreases nationally, there remains a very 
costly set of errors that represent a relatively small percentage of total 
cases. Further, these errors are distributed heterogeneously, and they are 
elusive to methods that have been effective in the past. FNS has sponsored 
several initiatives to reduce errors. One of the most significant resources 
for error reduction strategies is state and local food stamp agencies. By 
virtue of having responsibility for service delivery, state and local agencies 
have firsthand knowledge of the systems that allow errors to occur. This 
knowledge cannot be duplicated within FNS or by external sources. 

In July of 1983, USDA solicited state and local agencies to submit 
proposals for demonstrations and evaluations designed to reduce errors and 
abuse in the Food Stamp Program through fraud prevention and/or detection 
strategies and improved management practices. Authorization for this 
solicitation is contained in Section 17 of the Food Stamp Act, as amended. 
FNS established cooperative agreements with three states — Maryland, North 
Carolina, and Vermont— as a result of this solicitation. This summary 
presents a description of the three projects and their outcomes. 



1 



Title Page 



MARYLAND: A MEDIA INTERVENTION 

Maryland looked to its quality control data to reveal that client error 
was to be the broad target of the demonstration, and that income and household 
size were the major error sources. Maryland postulated that applicants do not 
intentionally misreport to cheat the system, but that they misreport because 
of fear that more complete and accurate reporting would reduce grants or 
render them ineligible. 

Given this framework, the most obvious correction action strategy is that 
of allaying client fears by informing clients that accurate reporting will not 
have negative consequences. The intervention to be demonstrated was 
conceptualized in a research office and developed in advertising terms, as 
media messages intended to foster accurate reporting. The media messages were 
designed to reinforce existing reporting requirements, and to make it clear 
that complete reporting does not necessarily have negative consequences. 

Although this approach had a logical basis, it was incomplete in that it 
ignored the fact that reporting more may well lead to receiving less. This 
perspective, which is probably obvious to local office staff (who were not 
consulted until the project was underway), did not enter into plans. Hence, 
there was a question of the face validity of the intervention from the onset. 

Maryland awarded a subcontract to an advertising firm, which competently 
developed a video and a brochure to convey the message defined by the project 
staff. The brochure was used as a stand-alone device, and as a reinforcement 
to the video. 

The demonstration was conducted in six local offices in Baltimore City, 
Montgomery County, and Prince George's County. The implementation proceeded 
fairly smoothly. However, it was noted that the video became obtrusive in 
busy waiting rooms as it played over and over before the same clients. 

The evaluation design used experimental and control groups at each site. 
Exposure to the interventions were made at randomly selected times. Although 
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it was desirable to balance the client mix (e.g., applicants, reappl icants , 
recertifications) , it was not operationally feasible to do so. 

Since the intervention targeted income and household composition, the 
desired outcome measures were errors in grant amounts attributed to those 
sources. This was expanded to include motor vehicle ownership. Although 
desirable to conduct QC-like verifications to detect impact on these types of 
errors, the Maryland budget on data collection was not sufficient to pursue 
this avenue. Instead, it was decided more practical to look to other existing 
data in the state (e.g., motor vehicle files, state income tax files, etc.) as 
verification sources. Without a sophisticated MIS, the Maryland project had 
no cost efficient way to compute the effect of discrepancies on grants. 
Hence, outcome measures were limited to differences in verified values among 
the treatment and control groups. 

Although a plausible approach, this plan broke down in implementation. 
The office conducting the research did not have previous linkages and working 
relationships with the other state agencies that would have to support the 
effort. Nor were they very successful in developing them. This became 
evident in a pilot test of the data gathering procedures, along with another 
problem. Copies of applications were very slow in coming from the local 
offices, and once they arrived, data were sparse and sketchy. This was 
particularly true for recertifications and clients who also received public 
assi stance. 

Both of these problem areas resulted in far more elapsed time spent 
attempting to create a data base. The verification of data was of 
questionable quality because of problems in matching identifiers, the time lag 
between verification sources and reported data, and the failure of 
verification information to address the desired outcome measures. Further, 
unexpected outcomes occurred — more motor vehicles were reported to the Food 
Stamp Program than were registered by the state. 

Because of the large consumption of time and resources associated with 
data collection, fewer resources than were planned were available for 
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analysis. The analyses that were conducted did follow the analysis plan. 
However, time and budget permitted only a "first cut" of analysis instead of a 
full investigation of the findings as they emerged. The project report does 
not take a stand on whether the interventions reduced error. Rather, it 
states that the findings are inconclusive. Because key information was not 
included in the report, it is not possible to determine the effect of the 
demonstration. Maryland did provide an analytic file to FNS so that a more 
complete analysis could be undertaken. 

Despite the weaknesses in the data and the analysis, Maryland did provide 
two products of quality from the perspective of communications technology. 
The project report, however, did not indicate whether Maryland planned to 
continue their use. The video and brochure are available, nevertheless, to 
other states that perceive errors occur because applicants believe that 
complete reporting will render them ineligible. 

NORTH CAROLINA: A COMPUTER-ASSISTED APPLICATION 

The rationale for North Carolina's selection of an intervention was that 
errors occurred because eligibility workers were not consistent and thorough 
during application intake. North Carolina proposed to improve the quality of 
interview data by the use of more structured interviewing modalities: A 
Structured Manual Interview (SMI) and a Computer-Assisted Interview (CAI). 
These products were developed by the Center for Urban Affairs and Community 
Studies (CUACS) at the University of North Carolina, under contract to the 
State of North Carolina. 

Using a commercial data base manager, the CAI was developed for use on 
personal computers. There were inadequacies in the software's capacity for 
human engineering and user friendliness in this application, however. In this 
instance, the choice of a prototyping approach to software development proved 
to be inadequate. In retrospect, it would have been more desirable to use a 
requirements definition approach, which would have more readily revealed the 
software capacity. As a consequence, development of the products, 
particularly of the CAI, was time-consuming and fraught with problems. More 
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planning at the outset would have been very beneficial. CUACS did a good job, 
however, of involving state and local staff at all stages of development, and 
eventually produced a CAI that is at least workable and acceptable to local 
staff. The SMI, while "workable" in the strict sense of the word, is 
extremely time consuming and is probably not useful except as a training tool. 

The research design associated with this effort had several positive 
features; it included random assignment of workers (within three counties) to 
experimental and control conditions, and use of pre-implementation and 
post-implementation scores. These scores were based on abstracting case files 
to develop measures of extent of reporting, of completeness of the application 
form, and of proportion of documented verifications. The choice of these 
surrogates for case error was made because the early sample sizes needed to 
detect reductions in case errors far exceeded the project budget and 
implementation feasibility. 

While the development of the interventions was problematic, implementation 
of the demonstration and the evaluation proceeded fairly smoothly. CUACS 
staff were effective in gaining cooperation from participating caseworkers in 
three counties, and developed and conducted a comprehensive training program. 
They were also effective in gathering the data according to plan, but placed a 
great deal of emphasis on collecting a variety of work measurement data. 
These data were mostly discarded in analysis, due to their questionable 
qual i ty. 

North Carolina did succeed in conducting its analysis according to plan. 
Results were presented as comparisons of ratios and composites on the 
dimensions of efficiency, effectiveness, and both. These were difficult to 
interpret in themselves and impossible to interpret relative to reductions in 
case errors. Nevertheless, it appears that the CAI has a beneficial effect on 
data quality, at least on the three dimensions that were analyzed, and that 
this effect is more pronounced for new applicants, as opposed to reappli cants 
or recertification clients. 

CUACS devoted a great amount of effort to studying the amount and distri- 
bution of worker time associated with each interview modality. It appears 
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that CAI interview time is not much greater than traditional interview time, 
and that the CAI leads to less time being spent on verification activities. 
Overall, the CAI is quite feasible in terms of the Food Stamp Program 
requirements, and is available to other states who choose to adopt North 
Carolina's application requirements. 

VERMONT: A PERFORMANCE MONITORING SYSTEM 

The Vermont demonstration originally intended to reduce agency error and 
client error through four interventions: 

• Supervisory case reviews (SCR) 

• Staff training program 
Quality circles (QC) 

• Performance objectives. 

These interventions were developed through analysis of Vermont's QC data, and 
as an extension and continuation of previous error reduction efforts. 
Further, these treatments were viewed as being interrelated, with each serving 
as a foundation for the next. During the course of the Performance Evaluation 
and Error Reduction Project (PEER), the training program was designed and 
partially implemented. However, it was concluded that the training seemed 
redundant relative to other PEER efforts, and was viewed negatively by the 
caseworkers. Instead of expending additional resources to perfect the 
training, PEER management, with FNS concurrence, decided to focus resources on 
the other interventions. PEER management also chose to exclude performance 
objectives from the PEER project, but to develop performance objectives as an 
extension of the SCR at a later date. 

The demonstration team developed and implemented a sound research design 
for the SCR. The Quality Circles did not proceed entirely according to the 
research design, but they did proceed in accordance with the circle process. 
That is, the circles did not reach the point of testing and evaluating 
specific outcomes, but rather continued as a process. In conducting the 
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project, key state staff worked closely with a subcontractor, Policy Studies, 
Inc., to carry out the demonstration and the evaluation. 

Although the project schedule was delayed, it was largely attributed to 
re-focusing and reshaping the project design in ways that made the project 
more useful to the State and improved the quality of the effort. In general, 
the project was carried out in an effective manner, with treatment planning, 
development and implementation carried out as planned, and with evaluation 
data collection and analysis proceeding similarly. This is not to say that 
the PEER project had no problems. Rather, it is to say that the problems that 
occurred were handled competently. 

The research design for the PEER project was bounded by Vermont's small 
size and the fact that from a statistical perspective the error rate was low. 
The state was too small to produce a sample large enough to detect statis- 
tically significant reductions in error. Hence, while the SCR treatment was 
found to reduce errors, the reductions could not be shown to be statistically 
significant. The Quality Circles proceeded to the point of identifying 
problems and solutions, but the project ended before the solutions were 
evaluated. Hence, treatment effects were qualitative, and positive outcomes 
were discovered through personal interviews. 

This project served as a further step in Vermont's effort to reduce errors 
by focusing on the role of the eligibility worker in controlling errors. As a 
result of the project, Vermont has not only adopted the SCR, but has also 
proceeded with performance objectives. The state continued the Quality 
Circles, but did not reach a final conclusion about expanding them throughout 
the state. Because the SCR was integrated into Vermont's automated case 
processing system (ACCESS), the concept rather than the product is directly 
transportable to other states. It has been sufficiently documented to do so. 
Documentation of the Quality Circle process was also complete and comprehen- 
sive, and is available to other interested states. 
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COMPARATIVE ASSESSMENT 

Although the three state demonstrations funded by FNS shared the common 
goal of error reduction they differed in many important ways — in design, in 
execution and in outcome. This discussion reviews the similarities and 
differences among the three demonstrations, and relates them to the outcomes. 

Origin and Nature of the Intervention 

The three state demonstrations differed at the outset in how the state 
chose the treatments. In Vermont the treatment was selected as a continuation 
of building upon what the state had done, and the treatment was integrated 
into the state's operating systems. The North Carolina project was conceived 
by the subcontractor and accepted by the state. Unlike the Vermont demonstra- 
tion which formalized and refined an existing supervisory case review 
practice, the North Carolina demonstration was the computerization of the 
application process — the replacement of a key operational step. The Maryland 
demonstra- tion was conceived by the Welfare research director and did not 
involve a subcontractor in executing the demonstration or evaluation. Like 
Vermont's intervention, the Maryland intervention did not replace a key 
operational step, but extended existing practices by using a more 
sophisticated media approach to explain eligibility requirements. Overall, it 
appears that the interventions that created the least turbulence in existing 
local office operations were easier to implement. 

The demonstrations also differed in who they primarily affected. The 
Vermont demonstration entered the system at the level of local office 
supervisors. Supervisors were most burdened by the intervention. Their 
participation affected eligibility workers, and in turn affected clients. The 
North Carolina's computer aided interviews affected eligibility workers and 
clients directly. The primary burden of this intervention was placed on the 
caseworker. In Maryland, the media presentations were placed in waiting rooms 
and hence affected receptionists and clients. This intervention placed little 
burden on either, but its repetitiveness in busy offices did make it become 
obtrusive. In examining the ease of implementation, the intervention that 
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affected more senior state staff (supervisors) went most smoothly, and those 
that affected clients and eligibility workers directly were more difficult to 
implement. This may have occurred because the supervisors are more likely to 
view responsibility for error reduction as a part of their job than 
caseworkers or receptionists. 

A key aspect of the evaluation design was the specification of outcome 
measures. The three demonstrations differed in outcome measures because these 
were linked to the nature of the treatments and the research hypotheses. The 
supervisory case review was primarily a detection mechanism — it identified an 
error and sought to correct it. There was a strong and direct link between 
the treatment and the outcome measure, and in fact the outcome measurement was 
an aspect of the treatment. Detecting and correcting errors gave supervisors 
immediate and positive feedback and an immediately evident effect. The 
outcomes in Maryland and North Carolina were not as proximate to the 
treatment. In addition, their effect would not be likely to be known to the 
individuals most directly affected by the treatment. Both the Maryland and 
North Carolina strategies were designed to prevent errors from occurring, not 
to correct an existing error. In general, prevention strategies are thought 
to be highly desirable because they have broader impacts and may achieve 
greater savings. However, by definition, they are not proximate to impact and 
hence those most affected by the intervention do not receive positive 
reinforcement about the effect of the treatment. Another drawback is that it 
is more difficult to measure these effects. Not being proximate to outcomes, 
the prevention strategies are more susceptible to other factors making a 
contribution to an outcome. Hence, prevention strategies require more 
rigorous research controls. This, in turn, often translates into more complex 
research designs, larger samples, more control groups, and more difficult 
measurement and data collection. Both the Maryland and North Carolina 
analyses of outcomes eventually broke down due to problems involving outcome 
data weaknesses and/or insufficient samples. Vermont, with a more proximate 
measure and a built-in data collection system (ACCESS) was better able to 
achieve its analytic goals. 
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Sophistication of State Infrastructure 

Vermont can be viewed as a more mature state in its service delivery and 
management information systems. In the absence of an automated management 
information system, Maryland had to launch a substantial primary data collec- 
tion effort. While North Carolina does have automated support, the project's 
research needs outstripped the system's capacity and North Carolina staff also 
had to expend substantial effort in data abstraction and collection. Hence, 
the state's sophistication and maturity also played into the demonstration's 
success . 

Project Organization 

The three demonstrations differed significantly in staffing and organiza- 
tion. The Vermont demonstration was directed by a senior state manager and 
involved several key staff in developing and implementing the treatments. 
This organization represented a strong commitment to the project. Vermont was 
supported by a capable subcontractor that assisted where needed, but had 
primary responsibility for the evaluation. The North Carolina Project 
Director was also a senior manager. However, the leadership duties were 
delegated to a more junior individual hired specifically to provide day-to-day 
management. The North Carolina project did not evidence high level involve- 
ment from top management. Other state staff did not play key roles, and the 
state's subcontractor had primary responsibility for the project. Problems 
emerged when the state's day-to-day project manager had to oversee a much more 
senior subcontract director and mediate between FNS and the subcontractor. 

In Maryland, the project was located in the research office and was headed 
by a mid-level manager. No other regular state staff were assigned to the 
project. Instead, it was staffed by contract employees. Although the level 
of leadership in Maryland was adequate, the lack of integration between the 
research office and the other state offices that was needed to support the 
project hampered data collection and data quality. 
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Implications 

This discussion suggests that error reduction demonstration success is 
linked to the following features: 

• A strong involvement, commitment, and leadership by senior staff of 
the Food Stamp State Agency 

• Assignment of key state staff to the demonstration project 

• Integrating the intervention into ongoing systems and case processing 
procedures 

• Sophisticated management information capabilities 

• Demonstrated linkages and cooperation from other state and local 
agencies whose participation is required for implementation of the 
intervention and collection of data for the demonstration evaluation. 

This discussion also notes that detection interventions that are 
integrated into existing systems and affect supervisory office staff are more 
amenable to successful implementation and evaluation. This is not to say, 
however, that these types of interventions are most successful at error 
reduction. The implication is that more effort, more support, more time and a 
larger budget is probably necessary to carry out a prevention mechanism 
successfully. This suggests that in considering future demonstrations, FNS 
pursue one of two avenues: 

• Select proposals that show promise of implementation success; or 

• Be prepared to provide the resources necessary to take a good raw 
idea and work it into a quality design and implementation. 

Volume III discusses how FNS can structure the demonstrations to assure this 
success. 
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